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© An artificial neural network is provided using a 
modular, self-organizing approach wherein a sepa- 
rate neural field is contained within each module (12. 
14 16 18) for recognition and synthesis of particular 
characteristics of respective input and output signals 
thereby allowing several of these modules to be 
interconnected to perform a variety of °pe f f t,or *; 
The first output and second input of one module (17) 
is respectively coupled to the first input and second 
output of a second module (14) allowing each mod- 
ule to perform a bi-directional transformation of the 
information content of the first and second input 
signals for creating first and second output signals 
having different levels of information content with 
respect thereto. In the upward direction, the first low- 
level input signal of each module is systematically 
delayed to create a temporal spatial vector from 
which a lower frequency, high-level first output signal 
is provided symbolic of the incoming information 
content. Since the first output signal contains the 
same relevant information as the first input signal 
while operating at a lower frequency, the information 
content of the latter is said to be compressed into a 
first high-level output signal. In the downward direc- 
tion a second output signal having a low-level of 
.nformation content is synthesized from a second 
input 'signal having a high-level of informat.on con- 



tent The second input signal is the best prediction 
of the first output signal available from the knowl- 
edge base of the module, while similarly the second 
output signal is the prediction of the first input signal. 
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SPANN: SEQUENCE PROCESS.NG ARTIFICAL NEURAL NETWORK 



Field of the Invention 

«■ 1*1 P T* nt invention re,ates in 9 en ^' to neu- 
ral networks and, more particularly, to a bi-direo 
.onal. hierarchical neural network comprising a plu- 
rality of interconnected modules wherein each 
module builds upon predetermined levels of learn- 
T 9 " i,i0n by com P fe «ing the information 
content of upward propagating signals while de- 
compressing downward propagating signals for 
proving f ifSt and second output signals 'symbot 
of the respective incoming information. 

Background of the Invention 

A biological neuron is a single nerve cell re- 
spons.ve to stimuli through weighted inputs known 
as synapses. The weighted stimuli are summed 
and processed through a particular non-linearity 
associated with the neuron. The output of the neu- 
ron may then be connected to the synapses of 
other neurons forming an interconnection of neu- 
rons known as a neural network, the latter of which 
possesses certain desirable properties including 
he abihty to learn and recognize information pat 
terns in a parallel manner. 

Technologists have long studied the advanta- 
geous nature of the biological neuron in an attempt 
to emulate its behavior in electronic circuitry As 
•ndicated. some of the advantages of the neural 
network include the learning and recognition of 
patterns and shapes. The neural network may be 
taught a particular pattern and later be called upon 
to identify the pattern from a distorted facsimile of 
the same pattern. Unlike conventional signal pro- 
cessmg techniques wherein the solution is pro- 
grammed with a predetermined algorithm, the rec- 
ogn.t.on technique of the neural network may be 
learned through an iterative process of applyinq 
repeated samples of an input signal, comparing the 
output signal thereof with the desired response and 
adjusting the synaptic weights until the desired 
response is achieved. 

Another advantageous feature of the neural 
network is the inherent parallel processing offering 
real-t.me execution without the classic von 
Neumann bottleneck associated with conventional 
ser.al processing techniques, i.e.. digital signal pro- 
cessing and general microprocessor based applica- 
tions. In general, neural networks are well suited for 
applications wherein the solution algorithm is not 
known or difficult to implement, or where conven- 
tional processing techniques are too slow. Notably 

dev 1C es, thus, a certam degree of error is generally 



present in the output signal. 

Conventional neural networks typ ica „ y com . 

wheL^thr^r^ ° nen r * id 
wherem the network is designed with a particula 

5 application in mind. i.e.. visual pattern recognition 
or speech recognition. Such neural networks work 
well for learning and recognizing static patterns to 
prov.de a higher-level output signal as compared to 
the input signal thereof, but usually fail to provide 
'0 satisfactory results when confronted with a spatial 
input signal operating in a temporal environment 
Furthermore, conventional neural networks typically 
requ.re special organizational steps and new train- 
'"9 sets for each new apptication. The need for the 
'5 predefined, highly specialized neural structure and 
initialization data sets limits the versatility of the 
neural network. A modular approach would be 
more desirable wherein a single-level neural field 
could be designed with some functional flexibility 
20 whereby several modules may be easily intercon- 
nected to perform a variety of learning and recogni- 
tion operations. Moreover, it would also be desir- 
able for the neural network to re-synthesize the 
input signal from the output signal thereby re- 
?5 creating the original lower-level information 

Hence, there is a need for a hierarchical, self- 
organ.z.ng neural network for use in general pur- 
pose, bi-directional recognition and synthesis pro- 
cesses wherein a first input signal having a low- 
30 level of information content is recognized for pro- 
viding a first output signal having a comparatively 
high-level of information content while, in the op- 
posite direction, a second low-level output signal is 
synthesized from a second high-level input signal. 

Summary of the Invention 

Accordingly, an objective of the present inven- 
tion is to advance the art of neural networks. 
<o In one aspect of the invention provides a neural 
network responsive to first and second input sig- 
nals applied at first and second inputs for providing 
first and second output signals at first and second 
outputs symbolic of the information content thereof 
45 comprising: 

first means responsive to the first output signal and 
havmg an output for providing a signal symbolic of 
the prediction of the first input signal; 
second means having first and second outputs and 
so first and second inputs coupled for receiving the 
first input signal and said signal symbolic of the 
prediction of the first input signal respectively for 
mon.tor.ng the degree of correlation therebetween 
-vhere.n the first input signal is gated through to 
sa.d first output for a low degree of correlation or . 
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alternately said signal symbolic of the pred.ct.on o 
the first input signal is gated through to sa.d first 
and second outputs for a high degree of correla- 
tion, said second output of said second means 
being coupled to the second output; 
third means for progressively delaying sa.d first 
output signal of said second means to prov.de a 
spatial vector thereof at a plurality of tap pants; 

and t 
fourth means responsive to said spatial vector of 
said first output signal of said second means and 
to the second input signal for providing the first 
output signal symbolic of the information content of 
the first and second input signals. 

In another aspect the invention provides a 
method of compressing a input signal for providing 
an output signal symbolic of the information con- 
tent thereof wherein the output signal operates at a 
lower nominal frequency while maintaining the 
same relevant information content of the input sig- 
nal, comprising the steps of: 
developing a signal symbolic of the predion of 
the input signal; 

monitoring the degree of correlation between the 
input signal and said signal symbolic of the predic- 
tion of the input signal; 

delaying the input signal for a low degree of cor- 
relation or alternately said signal symbolic of the 
prediction of the input signal for a high degree of 
correlation for providing a spatial vector thereof at a 
plurality tap points; and 

transforming said spatial vector into the output sig- 
nal symbolic of the information content of the input 
signal. 

The present invention provides a neural net- 
work for recognizing an input signal having a low- 
level of information content and providing an output 
signal having a comparatively high-level of informa- 
tion content while developing a knowledge base 
therefrom, with a neural network for synthesizing a 
second output signal having a low-level of informa- 
tion content from a second high-level input signal 
through the knowledge base of the network. It 
provides a bi-directional neural network utilizing a 
hierarchical structure of interconnected modules 
wherein the first output and second input of one 
module are respectively coupled to the first input 
and second output of another module whereby the 
recognition differential per module may be reduced 
for increasing resolution thereof without limiting the 
overall knowledge base. There is provided a neural 
network responsive to first and second input sig- 
nals applied at first and second inputs for providing 
first and second output signals at first and second 
outputs symbolic of the information content thereof. 
The neural network comprises a first circuit respon- 
sive to the first output signal for providing a signal 
symbolic of the prediction of the first input signal. 



A second circuit includes first and second outputs 
and first and second inputs coupled for receiving 
the first input signal and the signal symbolic of the 
prediction of the first input signal, respectively, for 
5 monitoring the degree of correlation therebetween 
wherein the first input signal is gated through to the 
first output for a low degree of correlation or al- 
ternately the signal symbolic of the prediction of 
the first input signal is gated through to the first 
,o and second outputs for a high degree of correla- 
tion. The second output of the second circuit is 
also coupled to the second output of the neural 
network. A third circuit progressively delays the 
first output signal of the second circuit for providing 
,s a spatial vector thereof at a plurality of tap points, 
while a fourth circuit is responsive to the spatial 
vector of the first output signal of the second circuit 
and to the second input signal for synthesizing the 
first output signal symbolic of the information con- 
20 tent of the first and second input signals. 

In another aspect, the present invention com- 
prises a plurality of serially coupled modules each 
being responsive to first and second input signals 
applied at first and second inputs for providing first 
25 and second output signals symbolic of the informa- 
tion content thereof, wherein the first output and 
the second input of a first one of the plurality of 
serially coupled modules are respectively coupled 
to the first input and the second output of a second 
30 one of the plurality of serially coupled modules for 
increasing the recognition and synthesis capacity 
thereof. 

Brief Description of Drawings 
35 . FIG. 1 is a simplified block diagram illustrating 
the preferred embodiment of the present inven- 
tion; 

FIG. 2 is a sample waveform plot of the first 
input signal useful in the explanation of the 
40 preferred embodiment; 

FIG. 3 is a simplified block diagram illustrating 
additional detail of one block of FIG. 1; 
FIG. 4 is a pair of simplified block diagrams 
demonstrating alternate forms of a single neu- 

45 ron; 

FIG. 5 is a pair of schematic and block diagrams 
illustrating one embodiment of the non-adaptive 
and adaptive synapses of a neuron; 
FIG. 6 is a waveform plot of a nonlinear behavior 

50 of the summing junction of a neuron; 

FIG. 7 is a simplified block diagram illustrating 
additional detail of the confidence layer block of 
FIG. 3; 

FIG. 8 is a pair of waveform plots showing 
55 confidence and novel states of the confidence 
layer of FIG. 7; 

FIG. 9 is a simplified block diagram illustrating 
additional detail of the feature map block of FIG. 
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FIG. 10 is a waveform plot illustrating the synap- 
tic weighting distribution of the neuron field of 
the feature map of FIG. 9; 
FIG. 11 is a simplified block- diagram illustrating 
additional detail of another block of FIG. 1; 
FIG. 12 is a simplified block diagram snowing 
additional detail of the confidence layer block of 
FIG. 11; and 

FIG. 13 is a simplified block diagram illustrating 
additional detail of the feature map block of FIG. 

Detailed Description of the Preferred Embodiment 

The preferred embodiment of the present in- 
vention is a modular approach to an artificial neural 
network wherein a plurality of modules may be 
interconnected for performing a variety of oper- 
ations. Referring to FIG. 1, there is shown se- 
quence processing artificial neural network 
(SPANN) 10 comprising modules 12, 14, 16 and 18 
interconnected between input 20 and output 22 and 
host computer 24. Input 20 is coupled to the first 
input of module 12, while the first outputs of mod- 
ules 12, 14 and 16 are coupled to the first inputs of 
modules 14, 16 and 18, respectively, through con- 
ductors 26. 28 and 30. Likewise, the second out- 
puts of modules 14. 16 and 18 are coupled to the 
second inputs of modules 12, 14 and 16 through 
conductors 32, 34 and 36, respectively, and the 
second output of module 12 is coupled to output 
22. The first output of module 18 is coupled 
through host interface 46 via conductor 48 to the 
first input of host computer 24, while the first 
output of host computer 24 is coupled through host 
interface 46 to the first terminal of switching circuit 
50. The second terminal of switching circuit 50 is 
coupled to the second input of module 18 via 
conductor 52. while the third terminal is coupled to 
conductor 48. Host computer 24 also includes a 
second output coupled to the control input of 
switching circuit 50 for controlling the contact clo- 
sure thereof. 

The first output and second input of one mod- 
ule, say module 12. are respectively coupled to the 
first input and second output of a second module, 
say module 14, allowing each module to perform a 
bi-directional transformation of the information con- 
tent of the first and second input signals for creat- 
ing first and second output signals having different 
levels of information content with respect thereto. 
In the upward direction, the first low-level input 
signal of each module is systematically delayed for 
creating a temporal spatial vector from which a 
lower frequency, high-level first output signal is 
provided symbolic of the incoming information con- 
tent. The number of elements of the spatial vector 
of the first input signal and the temporal spacing 



therebetween determines the amount of history 
used to recognize the first output signal and. thus 
provides a simple adjustment for customizing the 
transformation within the module. Since the first 
s output signal contains the same relevant informa- 
tion as the first input signal while operating at a 
lower frequency, the information content of the 
latter is said to be compressed into a first high- 
level output signal. 
io In the downward direction, a second output 

signal having a low-level of information content is 
synthesized from a second input signal having a 
high-level of information content. Thus, even with- 
out an external "lowest-lever first input signal a 
75 second low-level output signal may be synthesized 
through the knowledge base of the module given a 
second high-level input signal, that is, the second 
low-level output signal is decompressed from the 
higher-level second input signal. For ideal opera- 
20 tion, the second input signal is a duplication of the 
first output signal, and the second output signal is 
a duplication of the first input signal; however 
more practically, the second input signal is the best 
prediction of the first output signal available from 
25 the knowledge base of the module and the second 
output signal is the prediction of the first input 
signal. Furthermore, the first high-level output sig- 
nal of one module is applied as the first low-level 
input signal of the next upward module, while the 
30 second high-level input of the one module receives 
the second low-level output signal of this "next 
upward module" thereby creating a module hierar- 
chy which provides a high degree of resolution per 
module as well as a large knowledge base albeit 
35 limited by the economics of the number of mod- 
ules in one neural system. 

Consider for example a speech recognition and 
synthesis application wherein the first input signal. 
V 20 . such as shown in FIG. 2, is applied at input 
<o 20. The first input signal may, for example, com- 
prise the output voltage waveform of a microphone 
(not shown) responsive to audible speech sound 
waves, wherein the amplitude and frequency be- 
havior of the output voltage thereof varies with 
45 phonic content of the speech. If the user should 
utter the words. "START FIRST SEQUENCE" into 
the microphone, the sound waves are converted to 
an electronic signal which is applied at input 20. 
For example, the amplitude and frequency variation 
so of the first input signal occurring between times t, 
and t 2 of FIG. 2 is the word "START", while the 
word "FIRST" occurs between times t 2 and t 3 . and 
the word "SEQUENCE" is between times t 3 and U. 
SPANN 10 is organized for learning the fre- 
55 quency and amplitude characteristics of the first 
input signal and providing a first output signal as 
product of the recognition process of the learned 
information. A low-level information sequence such 
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as the waveform plot of FIG. 2 may be learned and 
recoani«d through the module hierarchy or * - 
2 providing three sequential high-level outpu 
InaMone each for the three words) at the first 
output of module 18 representative of the .ntorma 
tion content of the low-level signal ove the fme 
period t, to U permitting host computer Jo ac 
according to the instructions conta.ned therein S' 
^aneously. the second output of each module 
feeds downward to the second .nput o each mod 
ule for synthesizing a second low-level output ^ 
nal from a second high-level input s.gnal through 
the knowledge base thereof. 

For an illustration of the recogn.tion and syn 
thesis process. SPANN 10 is divided into four 
hierarchical categories having a one-to-one mw- 
respondence with respect to modules 12-18. WUh n 
each category, the level of information content ,s 
increased over and above the previous cafi^ 
For example, the amplitude and frequency variation 
of the first input signal may be converted through 
module 12 to signals symbolic of primitive sounds, 
although not yet discernable as a vowel or con- 
sonant With respect to module 12. the waveform 
olot of FIG. 2 is considered the first low-level .nput 
signal and the signals symbolic of primitive sounds 
are the second output signal having a compara- 
tively higher-level of information content. bkew.se. 
the signals symbolic of the primitive sounds are 
transformed into signals representative of pho- 
nemes (now discernable as a vowel or consonant) 
through module 14. For module 14. the signals 
symbolic of primitive sounds are the first low-level 
input signal and the signals representee of pho- 
nemes are the first high-level output s.gnal Con- 
tinuing the upward propagation, the signals o pho- 
nemes are transformed through module € to the 
signals of syllables, and the signals of syllables are 
converted through module 18 to the signals sym- 
bolic of words. Hence, the module hierarchy is bu.lt 
upon having the first high-level output signal of one 
module being applied to the first low-level .nput of 
the next module for providing sequentially h.gher- 
levels of recognition discrimination, over and above 
the previous module. For the preferred embody 
ment. module 12 is the lowest level and module 18 
is the highest level. 

The process of converting from one level of 
information, say the signals of syllables to another 
level of information, say the signals of words, is 
learned and stored within one module, .n this ^case 
module 18. Since one module is assigned the 
responsibility for one level of information trans- 
formation, it is understood that additional modules 
may be added and the aforedescribed categones 
may be further sub-divided whereby the recogn.- 
tion differential between levels is reduced and. ac- 
cordingly, the resolution and ability to discrim.nate 



therebetween is increased. 

Turning to FIG. 3. module 12 is shown ,n 
greater detail. The first input of module 12 that ,s 
fnput 20. is coupled to the first input of confidence 
5 ayTr 60. while *e first output of confidence layer 
5 Ts coupled to the input of delay circuit 62 and to 
the first input of feature map 64 through conductor 
61. Likewise, the output of delay circuit 62 ,s coup- 
,ed to the input of delay circuit 66 and to the 
, 0 second input of feature map 64 via conductor 63 
while the output of delay circuit 66 is coupled to 
the input of delay circuit 68 and to the third -nput o 
feature map 64 through conductor 67. The outpu 
o, delay circuit 68 is coupled to the fourth .nput of 
, 5 feature map 64 through conductor 69. ^rmore 
the second input of module 12 ,s coupted ^ttMhe 
fifth input of feature map 64 v.a conductor 32^ The 
first output of feature map 64 is coupled to the first 
output of module 12 that is conductor 26. and he 
20 second output of feature map 64 is coupled 
through output neuron 70 to the second .nput of 
confidence layer 60. The second output ^ of con- 
fidence layer 60 is then coupled to output 22 and 
the third output of confidence layer 60 .s coupled 
2s via conductor 128 to feature map 64. 

Before launching into further detail of the block 
diagram of FIG. 3. an explanation of one fundamen- 
tal building block of SPANN 10. a single neuron 
may be helpful. A well known block diagram of 
3 o neuron 90 is shown in FIG. 4a including inputs 92 
94 and 96 which are coupled to the .nputs of 
synapses 98. 100 and 102. respectively, wh.le the 
outputs thereof are coupled to summing |unc ion 
104 Synapses 98, 100 and 102 are we.ght.ng 
35 circuits wherein the signals applied at the .nputs 
may be attenuated, passed without modi«.cat.on or 
even amplified depending upon the relative we.gh- 
ting associated therewith. Hence, the s.gnals ap- 
plied at inputs 92. 94 and 96 are weighted through 
40 synapses 98. 100 and 102 and processed through 
the predetermined, typically nonlinear, operation of 
summing junction 104. The weighted nonl.near 
combination of the input signals .s prov.ded 
through output conductor 106. 
, 5 Referring to FIG. 4b. neuron 108 is shown as a 

functional equivalent of neuron 90 wherein the s.g- 
na.s applied at inputs 110. 112 and 114 are we.ght- 
ed through synapses 116. 118 and 120. respec- 
lively and processed through the nonlinear opera- 
so tion of summing junction 122 and flow through 
output conductor 124. Summing junction 122 may. 
for example, be realized with an operational am- 
plifier having positive and negative inputs config- 
ured as a conventional summing amplifier. In add.- 
S5 tion. conductor 124 is coupled to synapses 116 
118 and 120. while oscillator 126 and conductor 
128 are coupled to neuron 108. each providing an 
input to aid in the learning process. The signal 
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flowing through conductor 128 is sourced from 
confidence layer 60 for providing an indication of 
the degree of novelty of the first input signal for 
module 12, while the output signal of oscillator 126 
determines the learning rate of neuron 108, and the 
lateral feedback from conductor 124 enables the 
training for high activity outputs. Further discussion 
of these signals is forthcoming! 

Notably, synapse 1 16 is labeled in FIG. 4b as a 
non-adaptive variety, and synapses 118 and 120 
are labeled as the adaptive variety; adaptive synap- 
ses are shown as a square element, and non- 
adaptive synapses are shown as a circle. The 
distinction lies with the weighting, wherein a non- 
adaptive synapse has a fixed weight, i.e. it may be 
realized as a resistor, while the weight of an adap- 
tive synapse is adjustable and generally has some 
non-volatile memory associated therewith. Conse- 
quently, the adaptive synapse is typically much 
more complex as well as being fundamental to the 
learning process. The illustration of neuron 108 in 
FIG. 4b shall be used hereafter as a basic building 
block of SPANN 10 since it tends to simplify the 
drawings containing many neurons. 

Referring to FIG. 5a, non-adaptive synapse 116 
is shown including resistor 130 coupled between 
input 110 and summing junction 122. Alternately, 
adaptive synapse 118 is shown in FIG. 5b including 
transistor 132 having a drain coupled to input 112 
and a source coupled to one input of summing 
junction 122. The gate of transistor 132 is coupled 
through scaling circuit 134 to the first (negative) 
input of comparator 136, while the drain of transis- 
tor 132 is coupled to the second (positive) input of 
the same. The output of comparator 136 is coupled 
to the control input of switching circuit 138 and 
through inverter 139 to the control input of switch- 
ing circuit 140. The conduction path of switching 
circuits 138 and 140 are serially coupled between 
power supply conductors 142 and 144, while the 
conduction path of switching circuit 146 is coupled 
between the interconnection of switching circuits 
138 and 140 and the gate of transistor 148, wherein 
the latter includes a floating gate for maintaining 
non-volatile memory. The source of transistor 148 
is coupled to power supply conductor 150, typically 
operating at ground potential, and the drain of the 
same is coupled to gate of transistor 132 and to 
source of diode-configured transistor 152. The 
drain and gate of transistor 152 are coupled to- 
gether to power supply conductor 154, typically 
operating at a positive potential, V 00 . Switching 
circuit 156 includes first and second conduction 
terminals coupled between oscillator 126 and the 
control input of switching circuit 146, respectively. 
Furthermore, the first and second inputs of multi- 
plier 1 58 are coupled to conductor 1 28 and output 
conductor 124 of neuron 108, respectively, while 



the output thereof is coupled to the control input of 
switching circuit 156, as shown in FIG. 5b. Synapse 
120 follows the same construction and operation as 
synapse 118. 

5 The signal applied at input 112 is weighted 

through the drain-source impedance of transistor 
132 as determined by the potential developed at 
the gate thereof, the latter of which is also scaled 
through scaling circuit 134 to adjust for difference 

w in nominal levels between the gate and drain vol- 
tages which are then applied to the first and sec- 
ond inputs of comparator 136, respectively. If the 
voltage at the drain of transistor 132 exceeds the 
scaled gate voltage, the output signal of compara- 

75 tor 136 goes high closing switching circuit 138, 
while the same output signal is inverted through 
inverter 139 causing switching circuit 140 to open. 
Alternately, if the voltage developed at the drain of 
transistor 132 falls below the scaled gate voltage, 

20 the output signal of comparator 1 36 goes low open- 
ing switching circuit 138 and closing switching cir- 
cuit 140 again via inverter 139. The potentials ap- 
plied at power supply conductors 142 and 144 are 
sufficient to alter the charge on the floating gate of 

25 transistor 148, typically +20 volts and -20 volts 
respectively. Hence, the output state of comparator 
136 selects +20 volts or -20 volts to be applied 
through switching circuit 146 to the gate of transis- 
tor 148. 

30 The control input of switching circuit 146 is 

coupled through the conduction path of switching 
circuit 156 to oscillator 126, the latter of which 
provides a pulse train operating at a predetermined 
frequency. The control input of switching circuit 

35 156 is responsive to the output signal of multiplier 
1 58, the first and second inputs of which are coup- 
led to conductors 124 and 128. As stated, the 
signal flowing through conductor 128 is sourced 
from confidence layer 60 and provides an indica- 

40 tion of the degree of novelty of the first input signal 
for module 12. For example, a high signal flowing 
through conductor 128 may represent the novel 
state wherein the input signal is new and unknown 
of module 12. Conversely, a low signal flowing 

45 through conductor 128 may express a confidence 
state wherein the first input signal is known of 
module 12. Novel and confidence states are further 
discussed in the text covering confidence layer 60. 
If the signals flowing in conductors 124 and 128 

50 are high, wherein the latter indicates that the non- 
linear sum of the weighted input signals of neuron 
108 is high, then the control input of switching 
circuit 156 is activated as the product of these 
signals exceeds the control threshold thereof. In 

55 other words, synapse 118 is allowed to learn if (1) 
the output signal of neuron 108 is active indicating 
that it is sensitive to the amplitude and frequency 
characteristics of the first input signal present at 
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the first input, and (2) confidence layer 60 conveys 
a novel state indicating unknown information is 
present in the first input signal for learning. 

Given the novel state of the confidence layer 
and a high output signal for neuron 108. switching 
circuit 156 is rendered conductive causing switch- 
ing circuit 146 to periodically close as determined 
by the frequency of the output signal of oscillator 
126 thereby applying pulses of +20 volts or -20 
volts to the floating gate of transistor 148 depend- 
ing upon the output signal of comparator 136. The 
duty cycle of the pulses is such that no one single 
pulse has a significant impact on the conductivity 
of transistor 148; however, learning continues under 
such conditions since, over a period of time, the 
potential developed at the drain of transistor 148 
adjusts the voltage at the gate of transistor 132 
such that the signal flowing through the dram- 
source conduction path of the latter is weighted 
according to the likelihood of such a signal being 
present. The voltage developed at the gate of tran- 
sistor 132 shifts toward equilibrium with the signal 
applied to the drain thereof such that the differen- 
tial voltage across the first and second inputs of 
comparator 136 is reduced toward zero. Hence, 
adaptive synapse 118 has learned the frequency 
and amplitude characteristics of the first input sig- 
nal applied at the drain of transistor 132 and stored 
the information as a charge on the floating gate of 
transistor 148 whereby a weight is established to 
favorably match an input signal with such char- 
acteristics. The rate of learning is determined via 
the frequency of the output signal of oscillator 126 
which is typically 100 KHz. However, it is the 
combination of the synaptic weights which deter- 
mines the filtering behavior of neuron 108 to var- 
ious input signals applied thereto. A low signal 
flowing through either conductor 124 or 128 drives 
the output signal of multiplier 158 low opening 
switching circuit 156 and inhibiting the periodic 
closure of switching circuit 146 and. correspond- 
ingly the learning process, thereby maintaining the 
present value of the synaptic weight of transistor 
132. 

The nontinearity of summing junction 122 is 
illustrated m FIG. 6 as a sigmoid. The signals 
applied at snouts 110. 112 and 114 (V„o, Vm 2 and 
V,,4) are weighted through synapses 116. 118 and 
120 (W,: 5 . Wm 8 and W120) and applied to the 
inputs of summing junction 122, respectively. The 
sum of W,-;Vmc + W.tiVua ♦ WiaoViM deter- 
mines the output signal flowing in conductor 124. 
V,-». according to FIG. 6. wherein the output signal 
is zero with no input signals and maximizes in the 
positive and negative limits of the summation as 
shown. 

With the typical neuron building block in place, 
the discussion returns to FIG. 3 and continues with 



confidence layer 60. which is shown in greater 
detail in FIG. 7. Input 20 is coupled through the 
first input of confidence layer 60 to the input of 
gain controllable amplifier 170. the output of which 
5 is coupled to the first non-adaptive synapse of 
neuron 174. The output of neuron 174 is coupled to 
conductor 61 that is the input of delay circuit 62. 
The output of output neuron 70 is coupled to the 
input of gain controllable amplifier 175. the output 
to of which is coupled to output 22 and to the second 
non-adaptive synapse of neuron 174. The inputs of 
gain controllable amplifiers 170 and 175 are also 
coupled to the first and second non-adaptive syn- 
apses of neuron 176. while the output of the latter 
, 5 is coupled to the input of time averaging circuit 
178 Typically, the first and second non-adaptive 
synapses of neuron 176 are equal in magnitude 
and opposite in sign. The absolute value of the 
output signal of neuron 176 is taken prior to sum- 
20 ming and averaging within time averaging circuit 
178 allowing neuron 176 to monitor positive and 
negative variation at the inputs thereof. A negative 
value synapse may be realized for example by 
routing the output terminal of resistor 130 to the 
25 negative input of the aforementioned operational 
amplifier of summing junction 122. The output of 
time averaging circuit 178 is coupled to the first 
non-adaptive synapse of neuron 180. while the 
output of neuron 180 is coupled to the control input 
30 of gain controllable amplifier 170 and to the first 
non-adaptive synapse of neuron 182. The output of 
neuron 182 is coupled to the control input of gain 
controllable amplifier 175 and to the second non- 
adaptive synapse of neuron 180. Typically, the 
35 second synapse of neuron 180 and the first syn- 
apse of neuron 182 are negative values. A bias 
voltage is applied to the second non-adaptive syn- 
apse of neuron 182 at input 184. The output of 
neuron 180 is also coupled through conductor 128 
40 to adaptive synapses of feature map 64, as de- 
scribed for neuron 108. Neuron 176 operates as a 
comparing circuit by monitoring the first input sig- 
nal applied at input 20 and the signal provided 
from output neuron 70 which, as will be shown, is 
45 the prediction of the first input signal, while the 
output signal of neuron 176 is the degree of cor- 
relation of the first input signal and the synthesized 
second output signal, that is the output signal of 
output neuron 70. 
so Recall that one objective of the present inven- 

tion is to recognize the first input signal over a 
period of time and provide an output signal sym- 
bolic of the information content thereof. Consider 
an example wherein the beginning sequence of a 
55 command, say the word "START" in the command 
string of FIG. 2 is not immediately recognized and 
the synthesized output signal of output neuron 70 
is thus uncorrelated with respect to the first input 
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signal. Since the input signals of gain controllable 
amplifiers 170 and 175 do not track one another 
the output signal of neuron 176 tends to develop 
non-zero value, possibly high frequency, depend- 
ing upon the frequency of the first input signal. 
Time averaging circuit 178 is provided for maintain- 
ing a running time average at the output thereof 
which is applied to the first synapse of neuron 180. 
Over a period of time, the output signal of time 
averaging circuit 178 increases to a high level 
because of the low degree of correlation between 
the first input signal and the synthesized output 
signal of output neuron 70. 

The combination of neuron 180 and 182 be- 
haves as a flipflop circuit with hysteresis. If the first 
and second synapses of neurons 180 and 182 are 
assigned equal magnitude weights, then the output 
signal of neuron 180 switches high when the output 
signal of time averaging circuit 178 exceeds a 
predetermined value, i.e. V max of FIG. 8. in excess 
of the bias voltage applied at input 184 to the 
second synapse of neuron 182. The high output 
signal of neuron 180 holds the output signal of 
neuron 182 low through its second negative syn- 
apse. In addition, the output signal of neuron 182 is 
sufficiently low as to not overcome the high signal 
applied at the first synapse of neuron 180. 

The high output signal of neuron 180 enables 
gain controllable amplifier 170, while the low output 
signal of neuron 182 disables gain controllable am- 
plifier 175. This defines the novel state of con- 
fidence layer 60. wherein the first input signal flows 
through gain controllable amplifier 170 and neuron 
174 as well as delay circuits 62, 66 and 68 into 
feature map 64 where it may be recognized, and 
learned as necessary; see the aforedescribed op- 
eration of synapse 118. 

Once the amplitude and frequency behavior of 
the first input signal is recognized within feature 
map 64, the output signal of output neuron 70 
begins to track the first input signal thereby devel- 
oping a higher degree of correlation between these 
two signals causing the output signal of neuron 176 
to reduce toward zero indicating that module 12 is 
providing a high degree of predictability of the first 
input signal. Again, over a period of time, the 
output signal of time averaging circuit 178 de- 
creases toward zero and, as it drops to a predeter- 
mined level (V min of FIG. 8), the output signal of 
neuron 180 switches to a low state allowing the 
output signal of neuron 182 to switch to a high 
state whereby gain controllable amplifier 170 is 
disabled and gain controllable amplifier 175 is en- 
abled. The output signal of output neuron 70 may 
then pass through output 22 to an output device 
such as a speaker (not shown). Since the output 
signal of output neuron 70 is providing a high 
degree of predictability of the first input signal, the 



output signal of gain controllable amplifier 175 is 
applied to the second non-adaptive input of neuron 
174 to continue feeding feature map 64 with the 
prediction of the first input signal, while the con- 
5 duction path through gain controllable amplifier 170 
is disabled. The signal flowing in output 22. at least 
while confidence layer 60 is registering the con- 
fidence state, maintains a high degree of correla- 
tion with respect to the first input signal. Hence, the 
w second output signal of module 12 is a synthesized 
replica of the first input signal. 

As long as the output signal of output neuron 
70 tracks the first input signal, feature map 64 may 
continue to predict the next state of the first input 
is signal. This feature is key for providing the ability 
to recognize a known pattern of the first input 
signal even though in operation the first input sig- 
nal may be corrupted in the presence of noise. 
Once in the confidence state, the first input signal 
20 pattern symbolic of the letter "S" of the word 
"START" for example, may be recognized in spite 
of the noisy input signal. Time averaging circuit 
178 filters the instantaneous variation in the output 
signal of neuron 176 allowing feature map 64 to 
25 continue predicting as long as possible. However, if 
the output signal of output neuron 70 continues to 
. deviate from the first input signal over a period of 
time, the novel state is again declared by neuron 
176. 

30 The time constant of time averaging circuit 1 78 

limits the rate of change of confidence layer 60 
between the novel and confidence states and, con- 
sequently, it is established in accordance with the 
maximum estimated frequency component of the 
35 first input signal. For example, the time constant of 
time averaging circuit 178 may be 100 micro- 
seconds for a maximum frequency of 20 KHz for 
the acoustic input signal applied at input 20. The 
waveform plot of the output signal of time averag- 
40 ing circuit 178. V> 78 , and the output signal of 
neuron 180, Vi 8 o, are illustrated in FIG. 8. At time 
to. the output signal of output neuron 70 is tracking 
the first input signal and confidence layer 60 is 
registering a confidence state. At time t,. the first 
45 input signal changes behavior possibly in connec- 
tion with the beginning of the next word, say 
"FIRST", in the command string of FIG. 2. Shortly 
thereafter, as signal V 178 crosses the V max thresh- 
old, signal V, 80 switches to the novel state since 
so feature map 64 has not yet recognized the new 
amplitude and frequency characteristics of the first 
input signal. Although signal Vi 78 dips below the 
V ma ,c threshold at time t 2 . the output signal does 
not change state because of the hysteresis of neu- 
55 rons 180 and 182 until time t 3 at which time the 
output signal of output neuron 70 is on track with 
the first input signal, that is. the first input signal is 
recognized via feature map 64. driving the output 
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signal of neuron 176 low and reducing signal v,ti 
wSreby confidence layer 60 returns to the con- 

fidence state. t 
Continuing with FIG. 3. the signal provided at 
the first output of confidence layer 60 is systemati- 
cally delayed through delay arcu.ts 62 66 and 68 
for providing a spatial vector of the first mput signal 
wherein each element thereof is equal to the value 
of the first input signal at progressive later .n- 
stances in time. For example, assuming cons an 
and equal delays, say V seconds the output 
signals of delay circuits 62, 66 and 68 are V 6 o(to- 
J, V S o(to-2ta) and V 60 (to-3W). respectively, where 
V 60 (to) is the non-delayed time value. Hence, the 
spatial vector comprising elements V 60 (to). V S o(to- 
t,) Vsotfo^) and VsoOo-at,,) is applied through 
conductors 61. 63. 67 and 69 to the first, second, 
third and fourth inputs of feature map 64. respec- 
tively The length of the spatial vector is typically 
made much longer in practice, say 20 or more 
elements, with additional delay circuits thereby in- 
creasing the sample space of the first input signal. 
Typically, the time delay t d is 10 microseconds for 
processing voice data at a maximum frequency of 

20 KHz. _ , . 

Turning to FIG. 9. feature map 64 and output 
neuron 70 are shown in greater detail including a 
field of neurons 190. 192. 194. 196. 198. 200 202^ 
204 206 208. 210, 212. 214, 216. 218 and 220 
each having a first adaptive synapse coupled to 
conductor 61 for receiving signal Vso(to). Likew.se. 
conductor 63 is coupled to the second adaptive 
synapses of neurons 190-220. conductor 67 .s 
coupled to the third adaptive synapses of neurons 
190-220 and conductor 69 is coupled to the fourth 
adaptive synapses of the same. The outputs of 
neurons 190. 192, 194 and 196 are coupled to the 
first second, third and fourth non-adaptive synap- 
ses'of zone sensor neuron 222. and the outputs of 
neurons 198. 200, 202 and 204 are coupled to the 
first second, third and fourth non-adaptive synap- 
ses' of zone sensor neuron 224. respectively. In 
addition, the outputs of neurons 206, 208. 210 and 
212 are coupled to the first, second, third and 
, fourth non-adaptive synapses of zone sensor neu- 
ron 226. and the outputs of neurons 214, 216. 218 
and 220 are coupled to the first, second, third and 
fourth non-adaptive synapses of zone sensor neu- 
ron 228 respectively. The non-adaptive synapses 
of zone sensor neurons 222-228 are typically equal 
in weight. Conductors 26 and 32 each comprise 
four separate channels whereby the outputs of 
zone sensor neurons 222-228 are coupled to the 
four channels of conductor 26. It is understood that 
the recognition of the aforedescribed output signal 
symbolic of the information content of the spatial 
vector is provided at the outputs of neurons 190- 
220 and the use of zone sensor neurons 222-228 is 



optional; however, practically speaking, zone sen- 
sor neurons 222-228 are desirable since they com- 
bine the numerous output signals of neurons 190- 
220 thereby simplifying the interface between moo- 

5 U ' eS The sixteen outputs of neurons 190-220 are 
also respectively coupled to sixteen adaptive syn- 
apses of output neuron 70 as shown, wherem each 
of the weights thereof are equal to the respective 
, 0 adaptive synapses of neurons 190-220 correspond- 
ing to conductor 61. The magnitude of the output 
signal of output neuron 70 is proportional to the 
weighted sum of the output signals of neurons 190- 
220 hence, the former is the prediction of the 
, 5 signal flowing through conductor 61. Those output 
signals of neurons 190-220 having the greatest 
magnitude determine the prediction at the output of 
neuron 70 along with the adaptive weights driven 
by the most active set of neurons. Since the adap- 
20 tive synapses of output neuron 70 match the adap- 
tive synapses of neurons 190-220 which are coup- 
led to non-delayed conductor 61. the output s.gna 
of output neuron 70 is the prediction of the first 
input signal as determined from the knowledge 
2S base of feature map 64. The output of outpu 
neuron 70 is then coupled to the second input of 
confidence layer 60. 

The four channels of conductor 32 are coupled 
to the non-adaptive synapses of zone effector neu- 
30 rons 230. 232. 234 and 236. the outputs of which 
are coupled to non-adaptive synapses of neurons 
190-196. 198-204. 206-212 and 214-220. respec- 
tively The non-adaptive synapses of zone effector 
neurons 230-236 are also typically equal in weight 
3 g Furthermore, the outputs of neurons 190-220 

are coupled to one non-adaptive synapse of each 
neuron, one of which is shown for example m FIG- 
9 wherein the output of neuron 198 is coupled to 
one non-adaptive synapse of each of neurons 190- 
40 220. Likewise, the output of neuron 200 is coupled 
to non-adaptive synapses of neurons 190-220 (not 
shown), and so on. Hence, each neuron comprises 
a number of adaptive synapses equal to the num- 
ber of elements in the spatial vector, four in the 
45 present example; and a number of non-adaptive 
synapses equal to the total number of neurons in 
the field plus one for the associated zone effector 
neuron, totaling seventeen in the present example. 
The weighting assigned to the non-adaptive synap- 
se ses of each neuron in the field coupled to the 
output of the other neurons in the field follow the 
oraph shown in FIG. 10 which illustrates the associ- 
ative weighting of the adaptive synapses within a 
particular neuron. The construction and operation 
ss of neurons 190-236 follow that of neuron 108. 

One neuron, say neuron 198 for example, .s 
assigned the task of identifying a particular char- 
acteristic of the first input signal, say a change in 
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amplitude from a first level to a second level within 
a given period of time. Neuron 198 is typically 
grouped w,th other neurons, say neurons 200-204 
which identify other closely related characteristics 
of the f,rs : mput signal, and as such are considered 
to be , n the close proximity of. or equivalently. in 
he log,cal nefghborhood of neuron 198. Converse- 
ly, other neurons, say neurons 190-196. may iden- 
Wy characteristics of the first input signal much 
different than that of neurons 198-204. Hence ac- 
cording to FIG. 10. the weights of the non-adaptive 
synapses of neuron 198 coupled to the outputs of 
neurons 198-204 are set to a high value, i.e. a low 
impedance for resistor 130 of synapse 116, while 
the non-adaptive synapses of neuron 198 coupled 
to the outputs of neurons farther away in a logical 
sense, such as neurons 190-196. are set to a lower 
value again according to FIG. 10. Ultimately, the 
non-adaptive synapses of neuron 198 coupled to 
the outputs of neurons farthest away, say neurons 
«Ji 4-220. are assigned small, possibly negative val- 
ues. 

To better understand the filtering action of neu- 
ron 198 that provides the aforedescribed recogni- 
tion property, consider a simple training example 
wherein no apriori knowledge is stored in feature 

The adaptive synaptic wei9hts of neu ™s 

190-220 are randomized as is a common initializa- 
tion step known in the art of neuron networks The 
first input signal applied at input 20 alternates be- 
tween values of + 1 and -1 every V seconds. 
Since no apriori knowledge is provided, confidence 
layer 60 switches to the novel state allowing feature 
map 64 to learn the pattern. Oelay circuits 62 66 
and 68 are conveniently selected with delays of 
t« seconds so as to generate a spatial vector [1 
-1. 1.-1]. Assume that the adaptive synapses of 
neuron 198 by chance have been initially random- 
ized closest to the first input signal pattern [i -i i 
-1] thereby providing the highest activity at the 
output thereof encouraging neuron 198 to further 
learn the pattern, see discussion of adaptive syn- 
apse training. After numerous passes of the first 
input signal, the weights of the adaptive synapses 
of neuron 198 migrate toward the values [1, -i 1 
-1] thereby matching the characteristics of the first 
•nput Sl gnal over the given period. The first input 
signal has thus been learned and neuron 198 now 
possess the ability to recognize such a pattern 
Further assume that a another neuron, say neuron 
200. has learned the first input signal pattern [1.1. 

The first input signal sequence [1. -i 1 -ji IS 
now applied to input 20 and. consequently, to the 
adaptive synapses of neurons 190-220. The output 
signal of neuron 200. V 20 o. is equal to: 



V¥ 200.4Vgg) (1) 

- '((DO) + (D(-1) + (i)(D ♦ (-dm,) 

- o(2) 

s 

where: 

°( ) is the nonlinear transfer function (sigmoid) of 
neuron 200 ; 

Waoo.i is the value of the first adaptive synapse of 
10 neuron 200 

Wwo.2 is the value of the second adaptive synapse 
of neuron 200 

Wjoo.3 is the value of the third adaptive synapse of 
neuron 200 

'5 W M04 i S the value of the fourth adaptive synapse 
of neuron 200 

ductor eT V3lUe ° f Si9nal f ' 0Win9 thr0UQh COn " 

V 63 is the value of the signal flowing through con- 
20 ductor 63 y 

V 67 is the value of the signal flowing through con- 
ductor 67 

V 69 is the value of the signal flowing through con- 
ductor 69 

25 The output signal of neuron 1 98. V, 98 . is equal to: 
V, 98 

- °(W, 9M v SI + Wl98 . 2 v 63 ♦ w I98 3 v 67 + 

W 198.4V 69 ) (2) 
30 = o((1)(1) + (-1)(. 1)+ + ( . 1)( . 1M 

= o(4) 



as 



where: 

W, 98 ., is the value of the first adaptive synapse of 
neuron 198 

W, 982 is the value of the second adaptive synapse 
of neuron 198 

W, 983 is the value of the third adaptive synapse of 
neuron 198 

«o W, 98 . 4 is the value of the fourth adaptive synapse 
of neuron 198 

Comparing equations (1) and (2). the value of 
o(4) read from the sigmoid of FIG. 6 is greater than 
the value of o(2). hence, neuron 198 wins the 
45 competition and the aforedescribed characteristic 
of the first input signal is identified. In general the 
neuron or group of neurons having the largest 
magnitude of output signal is considered the win- 
ner of the competition over the input spatial vector 
so whereby the sample space of the first input signal 
is compressed into the first output signal having a 
higher-level of information content than that of the 
first input signal. By sampling the low-level input 
signal over a period of time and providing a si- 
55 multaneous vector representation thereof, the out- 
put signal of the winning neuron(s), neuron 198 in 
the present example, contains the same relevant 
information as contained in the sample space of 
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the first input signal. 

The transformation is possible, in part, through 
the spatial vector formed over multiple tone sam- 
ples of the first input signal whereby the neurons of 
feature map 64 can monitor the first input signal 
over a predetermined temporal sample space 
thereby providing the first output signal in real-time 
response thereto representative of the best predic- 
tion based upon the knowledge stored in the adap- 
tive synapses of the individual neurons. It is under- 
stood that feature map 64 in practice may com- 
prise many more neurons with many more adaptive 
and non-adaptive synapses than illustrated in FIG. 
9 and the spatial vector is typically much longer 
than the preceding example; however, a more in- 
volved system follows the same fundamental con- 
nection and operation, as described herein. The 
contribution of the zone effector neurons 230-236 
and the output signals of neurons 190-196 and 
200-220 applied to the non-adaptive synapses of 
neuron 198 have been momentarily neglected to 
simplify the example. In practice, zone effector 
neurons 230-236 bias neurons 190-220 through the 
respective non-adaptive synapse through the sec- 
ond high-level input signal from module 14. while 
the non-adaptive synapses are coupled to the re- 
spective outputs of neurons 190-220, wherein the 
weights of the latter are established in accordance 
with FIG. 10 for providing feedback signals which 
tend to favor the neuron winning the compet.t.on 
thereby creating a bubble of activity centered the 
winning neurons. 

By now it should be appreciated that one im- 
portant feature of delay circuits 62-68 is to provide 
a sample space of the first input signal over mul- 
tiple units of time. Through the spatial vector, the 
neurons of feature map 64 compress the informa- 
tion provided in the first input signal into the first 
output signal containing the same relevant informa- 
tion only at a lower operating frequency. The aver- 
age period of pattern stability of the sample space 
is compressed into a typically unique, continuous 
output pattern having a longer average period of 
pattern stability, that is. the first output signal is 
said to have a high-level of information content. 
Hence, the first output signal of module 12 flowing 
through conductor 26 to the first input of module 
14 may be symbolic of primitive sounds generated 
from the first input signal over the sample space of 
the spatial vector, and since the spatial vector is 
continuously updated with new information from the 
first input signal, the output signal of module 12 is 
the first input signal for module 14. albeit operating 
with a lower frequency than the first input s.gnal 
applied at input 20. 

Turning to FIG. 11. module 14 is shown in 
greater detail having a similar construction. as the 
aforedescribed module 12. but including some no- 



table distinctions. The first output of module 12 is 
coupled through conductor 26 which includes four 
channels from feature map 64 to the first input of 
module 14. the latter of which is also coupled to 
s the first input of confidence layer 240. Since, con- 
fidence layer 240 is processing four ind.v.dual 
channels, the first output thereof also provides four 
channels which are coupled to the inputs of delay 
circuits 242. 244. 246 and 248 and to feature map 
,0 250 via conductors 252. 254. 256 and 258. respec- 
tively Likewise, the outputs of delay circuits 242 
244 246 and 248 are coupled to the inputs of 
delay circuits 260. 262. 264 and 266 and through 
conductors 268. 270. 272 and 274 to feature map 
,s 250. while the outputs thereof are coupled to the 
inputs of de.ay circuits 276. 278. 280 1 and 282 and 
to feature map 250 via conductors 284. 286. 288 
and 300. respectively. The outputs of delay circuits 
276 278 280 and 282 are coupled through con- 
20 ductors 302, 304, 306 and 308 to feature map 250^ 
The first output of feature map 250 is coupled 
through conductor 28 to the first input of module 
16 and the second output of feature map 250 is 
coupled through output neuron 310 to the second 
25 input of confidence layer 240. The second output 
of confidence layer 240 is coupled through conduc- 
tor 32 to feature map 64 of module 12. Similarly, 
the second output of module 16 is coupled through 
the second input of module 14 to feature map 250. 
30 and the third output of confidence layer 240 is 
coupled via conductor 358 to feature map 250. 

The block diagram of confidence layer 240 is 
illustrated in FIG. 12 having four similar channels, 
one of which is illustrated in FIG. 7 and thoroughly 
3S described the associated text. The first channel of 
conductor 26 is coupled to the input of gain con- 
trollable amplifier 312. the output of which is coup- 
led to the first non-adaptive synapse of neuron 314 
The output of neuron 314 is coupled to the input of 
40 delay circuit 242 that is conductor 252. and the first 
output of output neuron 310 is coupled to the input 
of gain controllable amplifier 316. the output of 
which is coupled to the first channel of conductor 
32 and to the second non-adaptive synapse of 
45 neuron 314. The inputs of gain controllable amplifi- 
ers 312 and 316 are coupled to the first and 
second non-adaptive synapses of neuron 318. re- 
spectively, while the output of the latter is coupled 
to the first input of time averaging circuit 320. The 
so second channel of conductor 26 is coupled to the 
input of gain controllable amplifier 322. the output 
of which is coupled to the first non-adaptive syn- 
apse of neuron 324. The output of neuron 324 is 
coupled to the input of delay circuit 244 that .s 
55 conductor 254. and the second output of output 
neuron 310 is coupled to the input of gam control- 
lable amplifier 326. the output of which is coupled 
' to the second channel of conductor 32 and to the 
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second non-adaptive synapse of neuron 324 The 
•nputs of gain controllable amplifiers 322 and 326 
are coupled to the first and second non-adaptive 
synapses of neuron 328. respectively, while the 
output of the latter is coupled .to the second input s 
of time averaging circuit 320. The third channel of 
conductor 26 is coupled to the input of gain con- 
trollable amplifier 332, the output of which is coup- 
led to the first non-adaptive synapse of neuron 334 
The output of neuron 334 is coupled to the input of w 
delay circuit 246 that is conductor 256. and the 
second output of output neuron 310 is coupled to 
the input of gain controllable amplifier 336 the 
output of which is coupled to the third channel of 
conductor 32 and to the second non-adaptive syn- , 5 
apse of neuron 334. The inputs of gain controllable 
amplifiers 332 and 336 are coupled to the first and 
second non-adaptive synapses of neuron 338 re- 
spectively, while the output of the latter is coupled 
to the third input of time averaging circuit 320. The 20 
fourth channel of conductor 26 is coupled to the 
input of gain controllable amplifier 342. the output 
of which is coupled to the first non-adaptive syn- 
apse of neuron 344. The output of neuron 344 is 
coupled to the input of delay circuit 248 that is 25 
conductor 258. and the second output of output 
neuron 310 is coupled to the input of gain control- 
lable amplifier 346, the output of which is coupled 
to the fourth channel of conductor 32 and to the 
second non-adaptive synapse of neuron 344. The 30 
inputs of gain controllable amplifiers 342 and 346 
are coupled to the first and second non-adaptive 
synapses of neuron 348. respectively, while the 
output of the latter is coupled to the fourth input of 
time averaging circuit 320. 35 

The first and second non-adaptive synapses of 
neurons 318. 328. 338 and 348 are typically equal 
m magnitude and opposite in sign. The absolute 
value of the output signals of neurons 318, 328. 
338 and 348 are taken prior to summing and aver- 40 
aging within time averaging circuit 320. The output 
of time averaging circuit 320 is coupled to the first 
non-adaptive synapse of neuron 352. while the 
output of neuron 352 is coupled to the first non- 
adaptive synapse of neuron 354 and to the control « 
inputs of gain controllable amplifiers 312. 322. 332 
and 342. The output of neuron 354 is coupled to 
the second non-adaptive synapse of neuron 352 
and to the control inputs of gain controllable am- 
plifiers 316. 326. 336 and 346. Typically, the sec- so 
ond synapse of neuron 352 and the first synapse of 
neuron 354 are negative values. A bias voltage is 
applied to the second non-adaptive synapse of 
neuron 354 at input 356. The output of neuron 352 
is also coupled through conductor 358 to adaptive 55 
synapses of feature map 250, as described for 
neuron 108. 

The operation of confidence layer 240 follows 



21 °, f <; on,,denc e 'aver 64 wherein neurons 318. 
328. 338 and 348 independently monitor the re- 
spective correlations between the first input signals 
and the output signals of output neuron 310 where- 
by the degree of predictability of feature map 250 
may be time averaged and quantized into a con- 
fidence state and a novel state reflected through 
the outputs of neurons 352 and 354 for controlling 
the flow of information through module 14 as well 
as enabling and disabling the learning process of 
feature map 250 via conductor 358. 

In the novel state, as the output signal of time 
averaging circuit 320 increases to high level the 
output signals of neurons 352 and 354 switch to 
high and low levels, respectively, whereby gain 
controllable amplifiers 312. 322. 332 and 342 are 
enabled and gain controllable amplifiers 316. 326 
336 and 346 are disabled. The first input signals 
flow through neurons 314. 324. 334 and 344 and 
delay circuits 242-282 into feature map 250 for 
recognition, and learning as necessary. When the 
amplitude and frequency pattern of the first input 
signals are recognized, confidence layer 240 
switches to confidence state and the output signals 
of neurons 352 and 354 switch to low and high 
levels, respectively, whereby gain controllable am- 
plifiers 312. 322. 332 and 342 are disabled and 
gam controllable amplifiers 316, 326. 336 and 346 
are enabled. The output signals of output neuron 
310 may then pass through conductor 32 to zone 
effector neurons 230-236 of module 1 2. 

The signal provided at the first output of con- 
fidence layer 240 is systematically delayed through 
delay circuits 242. 260 and 276 for providing spa- 
tial vector [V 240 , (to ). Wto-y. V 240 .,(t 0 -2t d ) and 
v 2 4o. t (t 0 -3t d )J, where V 240 .,(to) j S the non-delayed 
time value of the signal Mowing through conductor 
252. Likewise, the second output signal of con- 
fidence layer 240 is delayed through delay circuits 
244, 262 and 278 for providing spatial vector 
tV 2 4 0 . 2 (t 0 ). V^o^to-t,,). Va^tfo^W and V nM (fa.3t i )- 
J. where V^^to) is the non-delayed time value of 
the signal flowing through conductor 254. The third 
output signal of confidence layer 240 is delayed 
through delay circuits 246. 264 and 280 for provid- 
ing spatial vector [V 240 . 2 ( W , v^t^). V 240 3{ x 0 -2t a ) 
and V24o. 3 (t 0 -3t d )J > where V 2403 (t 0 ) is the non-de- 
layed time value of the signal flowing through con- 
ductor 256. and the fourth output of confidence 
layer 240 is delayed through delay circuits 248. 
266 and 282 for providing spatial vector (V 2404 (t 0 )' 
Va-.o.ifto-W. V 2tf04 (t 0 -2t <f ) and V 240 . 4 (to-3y]. where 
V24o.«(to) is the non-delayed time value of the signal 
flowing through conductor 258. Since the maximum 
frequency of the first input signal of module 14 is 
lower than the maximum frequency of the first 
input signal of module 12. the delay through delay 
circuits 242-282 are made longer than those of 
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delay circuits 62-68. typically by a factor o four 
The increase in delay times between modules 12 
and 14 allows the latter to systematically increase 
the information content of the output signal per 
module. The average period of pattern stability of 
the sample space is compressed into a typically 
unique and continuous output pattern having a 
longer average period of pattern stability, that is 
the output signal is said to have a higher-level of 
information content. The transformation increases 
the pattern stability without losing the relevant in- 
formation content. 

Referring to FIG. 13. feature map 250 and 
output neuron 310 are shown in greater detail and 
follow the same construction and operation as de- 
scribed for feature map 64 and output neuron 70. 
More specifically, feature map 250 comprises a 
field of neurons 360, 362. 364. 366. 368. 370. 372 
374 376 378. 380. 382. 384. 386. 388 and 390 
each having a first adaptive synapse coupled to 
conductor 252 for receiving signal V 2 «.,(to). Like- 
wise, conductor 254 is coupled to the second 
adaptive synapses of neurons 360-390. conductor 
256 is coupled to the third adaptive synapses of 
neurons 360-390. and so on through conductor 308 
wherein the latter is coupled to the sixteenth adap- 
tive synapse of neuron 360-390 thereby completing 
sixteen connections per neuron. The outputs of 
neurons 360. 362. 364 and 366 are coupled to the 
first second, third and fourth non-adaptive synap- 
ses of zone sensor neuron 392. and the outputs of 
neurons 368. 370. 372 and 374 are coupled to the 
first second, third and fourth non-adaptive synap- 
ses of zone sensor neuron 394. respectively. In 
addition, the outputs of neurons 376. 378, 380 and 
382 are coupled to the first, second, third and 
fourth non-adaptive synapses of zone sensor neu- 
ron 396. and the outputs of neurons 384. 386, 388 
and 390 are coupled to the first, second, third and 
fourth non-adaptive synapses of zone sensor neu- 
ron 398. respectively. The outputs of zone sensor 
neurons 392-398 are coupled to conductor 28. 
Conductors 28 and 34 each comprise four separate 
channels. The sixteen outputs of neurons 360-390 
are also respectively coupled to the sixteen adap- 
tive synapses of output neurons 400. 402. 404 and 
406 as shown, wherein each of the weights of the 
latter are equal to the respective adaptive synapses 
of neurons 360-390 corresponding to conductor 
252. The outputs of output neurons 400-406 are 
coupled to the inputs of gain controllable amplifiers 
316 326, 336 and 346, respectively. The four 
channels of conductor 34 are coupled to the non- 
adaptive synapses of zone effector neurons 408. 
410. 412 and 414. the outputs of which are coupled 
to non-adaptive synapses of neurons 360-366. 368- 
374 376-382 and 384-390. respectively. 

The spatial vector provided at the tap points of 



delay circuits 242-282 allows the neurons of feature 
map 250 to compress the information provided 
through first input signal of module 14 into the 
second output signal thereof containing the same 
s relevant information only at a lower operating fre- 
quency. The amplitude and frequency variations of 
the first input signal over a period of time. i.e. the 
sample space, are transformed into a unique and 
continuous output signal containing the same rel- 
w evant information with a longer period of pattern 
stability as compared to the temporal sample 
space or. equivalent^, the first output signal con- 
tains a higher-level of information content. Thus, 
the first output signal of module 14 flowing through 
,s conductor 28 to the first input of module 16 may 
be symbolic of phonemes transformed from the 
first input signal over the sample space of the 
spatial vector symbolic of the primitive sounds^ 
Since the spatial vector is continuously updated 
20 with new information from the first input signal, the 
first output signal of module 14 is the first input 
signal of module 16 operating at a still lower fre- 
quency than that of the first input signal of module 

2S U Returning to FIG. 1. module 16 follows the 
same construction and operation as module 14 
feeding upon the first output signal thereof 
(phonemes) for providing an output signal having a 
high-level of information content (syllables). Uke- 
30 wise, module 1 8 follows the same construction and 
operation as module 14 receiving the first output 
signal from module 16 (syllables) and providing an 
even higher-level of information content (words) at 
the first output thereof. Accordingly, modules 16 
35 and 18 include delay circuits having delays greater 
than modules 14 and 16. respectively, typically by 
a factor of four between each module. 

It should be appreciated that the instructive 
connotations presented herein, i.e. primitive 
40 sounds, phonemes, syllables and words, are used 
to simplify the understanding of the present inven- 
tion- however, it is not necessary to divide the 
recognition and synthesis levels of modules 12-18 
upon such hard and precise boundaries. The accu- 
45 racy of the prediction and the degree of information 
compression is based upon the length of the spa- 
tial vector and the delay interval t*. and these 
parameters may be set at appropriate values com- 
mensurate with the design goals. 
50 The output signal of module 18 is applied 

through host interface 46 to the first input of host 
computer 24. Host interface 46 comprises an 
analog-to-digital converter and a digital-to-analog 
converter whereby the first input signals thereof are 
55 adapted for use by host computer 24. Simulta- 
neously, the digital signal provided at the first out- 
put of host computer 24 is coupled through host 
interface 46 to the first terminal of switching circuit 
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50. During normal recognition operation, the control 
signal provided at the second output of host com- 
puter 24 connects the second and third terminals 
of switching circuit 50 together whereby the first 
output signal with the highest level of information s 
content provided at the first output of module 18 is 
reapplied to the second input of the same via 
conductor 52. The other position of switching cir- 
cuit 50 is typically reserved for learning mode or 
dedicated synthesis mode whereby host computer w 
24 may supply the high-level information stored in 
a training file through conductor 52 to the second 
input of module 18. These training files in coopera- 
tion with the first input signal applied at input 20 
offer many advantages to simplify the initial training 15 
or re-training sequences. Furthermore, such train- 
ing files could re-configure the weights within mod- 
ules 12-18 to alter the utility application in a rapid 
and flexible manner as compared to convention 
manners such as described in the background of 20 
the invention. For example, SPANN 10 need not be 
dedicated as a voice recognition system, but rather 
could be automatically retrained as a visual rec- 
ognition system for patterns varying over time sim- 
ply by applying the appropriate training file. 25 

It is also possible with SPANN 10 to operate as 
a dedicated synthesizer. If, for example, the bias 
voltage applied to the confidence layer of each 
module, i.e. input 184 of module 12. was increased 
such that the confidence layer would continuously 30 
register the confidence state while high-level input 
signals were applied to the second input of module 
18. then the knowledge base of modules 12-18 
would systematically synthesize lower-level output 
signals ultimately providing a series of low-level 35 
signals at output 22 which could drive a speaker 
and audibly reproduce the high-level "words". Host 
computer 24 could easily store the high-level sig- 
nals while SPANN 10 systematically decompressed 
them through the module hierarchy for producing 4 0 
high quality audio output. For example, the signal 
symbolic of the word "START" could be applied to 
the second input of module 18 from which SPANN 
10 could generate the waveform of FIG. 2 between 
times t, and t 2 . 4$ 

It is further understood from the present exam- 
ple that the other module stacks may be intercon- 
nected with modules 12-18 in various configura- 
tions for providing more sophisticated recognition 2 
and synthesis capabilities. Indeed, the present in- 50 
vention is not limited to the speech recognition 
application as disclosed herein, but rather may 
encompass many applications wherein a waveform 
containing identifying features at different time 
scales must be learned for recognition and synthe- 55 

Sis. 

3. 

Hence, what has been provided is a novel 
modular approach to artificial neural networks com- 



prising a plurality of serially-coupled modules re- 
sponsive to first and second input signals wherein 
the first input signal of each module is systemati- 
cally delayed for creating a temporal spatial vector 
from which a lower frequency first output signal is 
provided symbolic of the information content of the 
first input signal. Simultaneously, the second high- 
level input signal is decompressed for providing a 
synthesized second low-level output signal corre- 
sponding to the information content of the second 
input signal. 

Claims 

1. A neural network responsive to first and sec- 
ond input signals applied at first (20) and sec- 
ond (32) inputs for providing first and second 
output signals at first (26) and second outputs 
(22) symbolic of the information content there- 
of, comprising: 

first means (70) responsive to the first output 
signal and having an output for providing a 
signal symbolic of the prediction of the first 
input signal; 

second means (60) having first (61) and sec- 
ond outputs (128) and first and second inputs 
coupled for receiving the first input signal and 
said signal symbolic of the prediction of the 
first input signal respectively for monitoring the 
degree of correlation therebetween wherein the 
first input signal is gated through to said first 
output for a low degree of correlation or al- 
ternately said signal symbolic of the prediction 
of the first input signal is gated through to said 
first and second outputs for a high degree of 
correlation, said second output of said second 
means being coupled to the second output; 
third means (62, 66, 68) for progressively* de- 
laying said first output signal of said second 
means to provide a spatial vector thereof at a 
plurality of tap points; and 
fourth means (64) responsive to said spatial 
vector of said first output signal of said second 
means and to the second input signal lor pro- 
viding the first output signal symbolic of the 
information content of the first and second 
input signals. 

The neural network of claim 1 wherein the first 
output signal symbolic of the information con- 
tent of the first and second input signals com- 
prises a plurality of output signals and the 
second input signal comprises a plurality of 
input signals. 

The neural network of claim 2 wherein said 
first means comprises a first neuron (70) hav- 
ing a plurality of inputs coupled for receiving 
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said plurality of outputs signals of the first 
output signal and having an output coupled to 
said second input of said second means (60). 

A neural network array comprising a plurality 
of serially coupled modules (12. 14, 16, 18) 
each being responsive to first and second in- 
put signals applied at first and second inputs 
for providing first and second output signals at 
first and second outputs symbolic of the in- 
formation content thereof, wherein the first out- 
put and the second input of a first one of the 
plurality of serially coupled modules are re- 
spectively coupled to the first input and the 
second output of a second one of the plurality 
of serially coupled modules for increasing the 
recognition capacity thereof, wherein each 
module comprises a network as claimed in any 
of claims 1 to 3. 

A method of compressing a input signal for 
providing an output signal symbolic of the in- 
formation content thereof wherein the output 
signal operates at a lower nominal frequency 
while maintaining the same relevant informa- 
tion content of the input signal, comprising the 
steps of: 

developing a signal symbolic of the prediction 
of the input signal; 

monitoring the degree of correlation between 
the input signal and said signal symbolic of the 
prediction of the input signal; 
delaying the input signal for a low degree of 
correlation or alternately said signal symbolic 
of the prediction of the input signal for a high 
degree of correlation for providing a spatial 
vector thereof at a plurality tap points; and 
transforming said spatial vector into the output 
signal symbolic of the information content of 
the input signal. 
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